Configure the audio session before recording starts, not only on connect - #1179
Draft
MaxHeimbrock wants to merge 1 commit into
Draft
Configure the audio session before recording starts, not only on connect#1179MaxHeimbrock wants to merge 1 commit into
MaxHeimbrock wants to merge 1 commit into
Conversation
On iOS the audio engine refuses to open the microphone unless the audio session already permits recording, and livekit_client owns that session: it disables flutter_webrtc's own session management at plugin registration. The policy was only pushed to native from Room.connect, so any recording that starts earlier ran against the app-default soloAmbient category and was rejected with kAudioEngineErrorAudioSessionInvalidCategory (-9001), surfacing as `AudioProcessingException(applyFailed): Audio engine returned error code: -9001`. That is the default path for agent sessions, since SessionOptions enables pre-connect audio, which buffers the microphone before connect and therefore failed on every attempt. A pre-join microphone preview hits the same wall without pre-connect audio involved (#1165). Push the policy from the capture path instead, where the microphone actually opens. AudioManager.prepareRecording() is a no-op once a policy has been pushed, so it never re-applies a configuration to a live session, and in manual mode, where the app owns the session. Fixes #1165 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hiroshihorie
added a commit
that referenced
this pull request
Sep 1, 2026
…pushed (#1182) ## Problem On iOS the WebRTC audio engine refuses to enable recording unless the audio session category permits input. Its pre-enable check returns `kAudioEngineErrorAudioSessionInvalidCategory` (-9001), which the SDK reported as `AudioProcessingException(applyFailed): Audio engine returned error code: -9001`. `livekit_client` owns the iOS audio session since #1108 (`LiveKitPlugin.swift` disables flutter_webrtc's session management at registration). The native engine observer (`LKAudioEngineObserver.willEnableEngine`) is the right hook and runs before the engine's check, but it only applied a configuration that Dart had pushed. In automatic mode the only push site was `Room.connect`, so anything that started recording earlier met an empty cache, the observer returned "proceed" with the session still `soloAmbient`, and the engine rolled back: - `Room.withPreConnectAudio`, which `SessionOptions.preConnectAudio` enables by default, so the Flutter agent starter failed on every "Start call" - a pre-join microphone preview (#1165) - an engine start driven from native before the Flutter side exists, for example the plugin's static `setEngineAvailability` on a CallKit killed-state wake Because the preconnect throw happens before `connect`, the cache was never seeded and the failure repeated on every attempt. ## How the Swift SDK handles this `AudioSessionEngineObserver.engineWillEnable` derives the session configuration from the requested engine state alone (`playAndRecord` presets while recording, `playback` for playout only), synchronously inside the engine's enable call. Nothing is configured "before connect". The engine asks, the observer configures, the engine starts. The Flutter plugin already has the same observer in the same place, it just had no built-in policy. ## Fix `LKAudioEngineObserver.effectiveConfigurationLocked` now resolves a built-in `playAndRecord` preset (`allowBluetooth | allowBluetoothA2DP | allowAirPlay`, `videoChat`) whenever nothing has been pushed and automatic management is on. The existing playout-only `playback` branch applies to it as well. Manual mode still leaves the session alone. The Dart-pushed policy becomes an override rather than a prerequisite, and for the default `AudioSessionOptions.communication` it pushes the same values, so the connect-time push does not change the live session. The preset is built on a copy of the shared `RTCAudioSessionConfiguration.webRTC()` object, and it is best-effort: if applying it fails, the engine start proceeds and the ADM's own pre-enable checks still gate recording, so apps whose own session was already valid are not newly rolled back with -4100. Only a policy the app actually pushed keeps failing the engine start hard. The mode is fixed to `videoChat` because it matches the Dart default speaker preference, and a non-default preference always arrives as a pushed policy whose mode already carries it, so the preset can never observe anything else. `LocalAudioTrack.startCapture` additionally pushes the resolved Dart policy to native before recording starts (cache-only while the engine is idle in automatic mode). Flutter-driven starts therefore always use the real Dart policy, and the built-in preset only stands in for engine starts that happen before the Flutter side exists, for example the plugin's static `setEngineAvailability` on a CallKit killed-state wake. The engine observer is also shared across plugin registrations now: a second Flutter engine registering in the same process (add-to-app, `FlutterEngineGroup`) previously reset the pushed policy and management mode silently, which the preset would have turned into an unwanted session activation. Only the notification channel is rebound per registration. ## Error mapping Audio device module results now get their own error codes on the `startLocalRecording`, `setEngineAvailability`, `setMicrophoneMuteMode` and `stopLocalRecording` channels, mirroring client-sdk-swift's `checkAdmResult`. A mute-mode change can rebuild the engine and hit the same pre-enable checks as a recording start, so all entry points surface the same failure the same way: | ADM result | Native error code | Dart exception | | --- | --- | --- | | -9000 `InsufficientDevicePermission` | `deviceAccessDenied` | `TrackCreateException` | | -9001 `AudioSessionInvalidCategory` | `audioSessionInvalidCategory` | `AudioSessionException` (new) | | -4100 `FailedToConfigureAudioSession` | `audioSessionConfigureFailed` | `AudioSessionException` (new) | | anything else | caller fallback (`applyFailed`, `setEngineAvailability`, ...) | unchanged | `AudioSessionException` is a new public class with its own changeset entry. ## Relation to #1179 @MaxHeimbrock's #1179 diagnosed this first and fixes it by pushing the policy from `LocalAudioTrack.startCapture` (`prepareRecording()`). This PR ends up covering both layers: `startCapture` pushes the resolved policy like #1179 does (without a Dart-side flag tracking native cache state), and the native observer is additionally self-sufficient like Swift's, so engine enables that never pass through Dart are covered too. With this merged, `prepareRecording()` becomes redundant. ## Testing - Flutter agent starter on an iPhone 17 Pro (iOS 27.0): unpatched, `Start call` failed with -9001 on every attempt. Patched, `startCapture()` succeeds two seconds before the connect-time policy push, the preconnect buffer is sent to the agent, and the call connects. - `flutter analyze`, `flutter test`, `dart format --set-exit-if-changed`, `import_sorter --exit-if-changed` clean. - New unit tests cover the error mapping. The iOS-gated branches are not reachable from unit tests (`lkPlatform()` has no seam), same limitation as #1179 noted. Fixes #1165 Refs #1042
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On iOS the audio engine refuses to open the microphone unless the audio session already permits recording, and livekit_client owns that session: it disables flutter_webrtc's own session management at plugin registration (
LiveKitPlugin.swift,setAudioSessionManagementEnabled(false)). But the policy was only pushed to native fromRoom.connect—NativeAudioManagement.start()→AudioManager.applyOptionsForConnect()→configureNativeAudio— so any recording that starts earlier ran against the app-defaultsoloAmbientcategory. The engine's pre-enable check rejects that withkAudioEngineErrorAudioSessionInvalidCategory(-9001), surfacing asAudioProcessingException(applyFailed): Audio engine returned error code: -9001.That is the default path for agent sessions:
SessionOptions.preConnectAudiodefaults to true, andwithPreConnectAudiobuffers the microphone before connect. BecausestartRecordingthrows beforeconnectis reached, the native policy cache is never seeded, so it fails on every attempt rather than only the first. A pre-join microphone preview hits the same wall with no pre-connect audio involved, which is how #1165 was reported.The fix pushes the policy from the capture path instead, where the microphone actually opens:
LocalAudioTrack.startCapturecalls the newAudioManager.prepareRecording()just beforeNative.startLocalRecording. That one choke point covers pre-connect buffering, a standalone pre-join track, and normal mic publish.prepareRecording()is deliberately narrow. It is a no-op off iOS, a no-op inAudioSessionManagementMode.manual(the app owns the session and is responsible for a recording-capable category), and a no-op once a policy has been pushed — tracked by a new_hasPushedAppleSessionPolicyflag set at both existing Apple push sites. That last guard is not just an optimization: re-pushing mid-call while the engine has playout only would resolve the playback category throughselectCategoryByEngineStateand apply it moments before recording starts, so the guard keeps this off the hot path entirely.The added unit test only asserts the off-iOS no-op —
lkPlatform()readsdart:io Platformwith no override hook, so no test in this repo can reach an iOS-gated branch.flutter analyze,flutter test,dart format --set-exit-if-changedandimport_sorter --exit-if-changedare all clean.How this went unnoticed
The precondition was never explicit.
PreConnectAudioBuffer(#830) opens the microphone outside the room lifecycle by design, and nothing stated who guarantees the session permits recording — it was satisfied incidentally, by flutter_webrtc'sensureAudioSessionongetUserMediaand by track-counting that fired on publish, which a pre-connect track has not reached.Two commits in 2.9.0, a day apart, removed that cover and then made the failure loud:
55af814) took sole ownership of the iOS session: it is wheresetAudioSessionManagementEnabled(false)first appears, killing flutter_webrtc's incidental configuration, and where track-counting became engine-driven lifecycle whose only Dart entry point is connect. After it, no path configures the session before connect.d1fe342) moved the ADM start intoLocalAudioTrack.startCaptureand made it fail-fast. Its description mentions the preconnect path, but the review lens there was audio processing options, so the session category beside it went unexamined. Combined with the fork's specific error codes, a latent misconfiguration became a named hard failure.The fail-fast was not the mistake — it exposed the real one. #1042 reports what looks like the same path failing at 2.7.0 with the generic
adm api failed with code: -1, five months earlier, which is likely the same root cause under a code too vague to act on.Two things kept it out of test range:
flutter build ios --release --no-codesign— a compile, with no simulator or device run.lkPlatform()readsdart:io Platformwith no seam, so on the test host every iOS-gated branch inAudioManageris unreachable. The existing audio tests cover policy resolution and channel argument encoding — pure Dart either side of the platform gate, never the gate itself.Worth considering separately from this PR: the native pre-enable check is gated
#if !TARGET_OS_OSX, so it runs on the simulator too — driving the example app through connect-with-mic there would have caught this end to end, and is the only layer that would have. A test seam forlkPlatform()would make these branches assertable at all, andwillEnableEnginefindingcachedConfigurationnil is worth logging loudly rather than silently returning proceed and letting the native check fail a layer later.Fixes #1165
Testing
I tested this locally with the Flutter Agent Starter: